Learning more with the same effort: how randomization improves the robustness of a robotic deep reinforcement learning agent

L. Güitta-López, J. Boal, A.J. López López

Applied Intelligence Vol. 53, nº. 12, pp. 14903 - 14917

Summary:

The industrial application of Deep Reinforcement Learning (DRL) is frequently slowed down due to an inability to generate the experience required to train the models. Collecting data often involves considerable time and financial outlays that can make it unaffordable. Fortunately, devices like robots can be trained with synthetic experience through virtual environments. With this approach, the problems of sample efficiency with artificial agents are mitigated, but another issue arises: the need to efficiently transfer the synthetic experience into the real world (sim-to-real).
This paper analyzes the robustness of a state-of-the-art sim-to-real technique known as Progressive Neural Networks (PNNs) and studies how adding diversity to the synthetic experience can complement it.
To better understand the drivers that lead to a lack of robustness, the robotic agent is still tested in a virtual environment to ensure total control on the divergence between the simulated and real models.
The results show that a PNN-like agent exhibits a substantial decrease in its robustness at the beginning of the real training phase. Randomizing specific variables during simulation-based training significantly mitigates this issue. The average increase in the model’s accuracy is around 25% when diversity is introduced in the training process. This improvement can translate into a decrease in the number of real experiences required for the same final robust performance. Notwithstanding, adding real experience to agents should still be beneficial, regardless of the quality of the virtual experience fed to the agent. The source code is available at: https://gitlab.com/comillas-cic/sim-to-real/pnn-dr.git

Spanish layman's summary:

Los agentes de Aprendizaje por Refuerzo Profundo entrenados con experiencia sintética se enfrentan a cómo transferir experiencia virtual a la realidad. Este artículo analiza la robustez de las Redes Neuronales Progresivas y estudia como añadir diversidad con experiencia sintética las complementan.

 

English layman's summary:

Deep Reinforcement Learning agents trained with synthetic experience face the challenge of how to transfer that knowledge to reality. This paper analyzes the Progressive Neural Networks’ robustness and studies how adding diversity with synthetic experience enhances the agent’s performance.

 

Keywords: Reinforcement Learning, Deep Learning, Sim-To-Real, Domain Randomization, Sample Efficiency, Robotics

JCR Impact Factor and WoS quartile: 5,300 - Q2 (2022)

DOI reference: https://doi.org/10.1007/s10489-022-04227-3

Published on paper: June 2023.

Published on-line: November 2022.

Citation:
L. Güitta-López, J. Boal, A.J. López López, Learning more with the same effort: how randomization improves the robustness of a robotic deep reinforcement learning agent. Applied Intelligence. Vol. 53, nº. 12, pp. 14903 - 14917, June 2023. [Online: November 2022]